Fix flaky TestStorageProviderUploadDownload integration test#21
Closed
Fix flaky TestStorageProviderUploadDownload integration test#21
Conversation
Exclude the test_logs directory created by test debugging scripts to prevent test artifacts from being committed to the repository.
Enhanced the Docker Compose setup for integration tests to handle transient failures and port conflicts: - Add retry logic (3 attempts) for starting services - Implement exponential backoff between retries - Add 2-second wait after cleanup for port release - Improve logging at each setup step This fixes flaky test failures caused by port conflicts when containers from previous test runs haven't fully cleaned up.
Enhanced test with detailed logging and verification steps: - Add timing information for each operation - Add object verification step before download - Add 100ms stabilization wait after upload - Add detailed error messages for data mismatches - Log connection details and test progress This logging helps diagnose flaky test behavior and provides better visibility into test execution for debugging purposes.
Add run_flaky_test.sh utility script that runs integration tests multiple times to help identify and debug flaky test behavior. Features: - Configurable number of test runs (default: 20) - Detailed logging of each run - Summary report with pass/fail statistics - Saves logs for failed tests for analysis This tool was used to diagnose and fix the flaky TestStorageProviderUploadDownload test.
Owner
Author
|
Superseded by #22 which integrates the stability testing methodology directly into the test suite rather than using an external bash script. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the flaky
TestStorageProviderUploadDownloadintegration test by improving Docker Compose setup, adding retry logic, and enhancing test logging.Problem
The integration test was failing due to port conflicts when Docker containers from previous test runs hadn't fully cleaned up. Port 5556 (used by the dex service) was being held by gvproxy even after
docker compose down, causing subsequent test runs to fail with "address already in use" errors.Test Results Before Fix:
Solution
1. Improved Docker Compose Setup
2. Enhanced Test Logging
3. Test Infrastructure
run_flaky_test.shutility script for debugging flaky teststest_logs/to.gitignoreTest Results After Fix
Verification:
Changes
testing/integration_test.go- Retry logic and cleanup improvementstesting/storage_integration_test.go- Comprehensive loggingrun_flaky_test.sh- New debugging utility.gitignore- Exclude test artifactsTesting
Run the test multiple times to verify stability:
Or use the included script: